CLAIMS 



We claim: 

\ 1. A method comprising: 

mitiating a search for images based on at least one query keyword in a 
query; anck 

identmdng, during the search, first images having associated keywords that 
match the quen^ keyword and second images that contain low-level features 
similar to those oMie first images. 

2. A method >as recited in claim 1, further comprising ranking the first 
and second images. \ 

3. A method as recite(i in claim 1, further comprising presenting the first 
and second images. \ 

4. A method as recited in claim 1 , further comprising: 
presenting the first and second imafees to a user; and 

monitoring feedback from the user\as to which of the first and second 
images are relevant to the query. \ 

5* A method as recited in claim 1, furtherVomprising: 
presenting the first and second images to a user\ 

receiving feedback firom the user as to whether tnfe first and second images 
are relevant to the query; and \ 
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learning how the first and second images are identified based on the 
feedbac^ from the user. 

6. Amethod as recited in claim 1, further comprising: 
presentingXhe first and second images to a user; 

receiving feedback from the user as to which of the first and second images 
are relevant to the query: and 

refining the searcmto identify additional images that contain low-level 
features similar to those of tft^ images indicated by the user as being relevant to 
the query. 

7. A method as recited in claim 1, further comprising: 
presenting the first and second images to a user; 
receiving feedback from the user as which of the first and second images 

are relevant to the query; and 

assigning a large weight to an associatio^between the query keyword and 
the images deemed relevant by the user. 

8. A method as recited in claim 7, further con^rising grouping the low- 
level features of the images deemed relevant by the user. 



9, A method as recited in claim 1 , further comprising 
presenting the first and second images to a user; 

receiving feedback from the user identifying an exampleN^mage as less 
relevant or irrelevant to the query for refinement of the search; and 
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\assigning a small weight to an association between the query kejword and 
the exarrole image. 

10. V method as recited in claim 9, further comprising identifying 
additional imageswwith low-level features similar to those of the example image. 

11. A computer readable medium having computer-executable 
instructions that, when Executed on a processor, perform the method as recited in 
claim 1 . \ 

12. A method comprising: 

permitting entry of both ke^vword-based queries and content-based queries; 

finding images using both\emantic-based image retrieval and low-level 
feature-based image retrieval; \ 

presenting the images to a user so that the user can indicate whether the 
images are relevant; and \ 

conducting semantic-based relevance fetedback and low-level feature-based 
relevance feedback in an integrated fashion. \ 

13. A method as recited in claim 12, further comprising ranking the 
images. \ 

14. A method as recited in claim 12, further comprising using images 
indicated as being relevant to find additional images. \ 
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\5. A computer readable medium having computer-executable 
instructions that, when executed on a processor, perform the method as recited in 
claim 12. \ 

16. A Vethod comprising: 

associating Veywords with images to form keyword-image links; 
assigning weights to the keyword-image links; 

presenting a redult set of images obtained from an image retrieval search 
based on a query; \ 

receiving feedback frbm a user as to whether the images in the result set are 
relevant to the query; and \ 

modifying the weights according to the user feedback. 

17. A method as recited in claim 16, wherein the modifying comprises 
increasing the weight of a keyword-imagedink for images deemed by the user as 
more relevant to the query. \ 

18. A method as recited in claim 16, wberein the modifying comprises 
decreasing the weight of a keyword-image link for images deemed by the user as 
less relevant to the query. \ 

19. A computer readable medium having Vomputer-executable 
instructions that, when executed on a processor, perform the ni^hod as recited in 
claim 16. \ 
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\20. A method comprising: 

prl^enting a result set of images that are retumed from an image retrieval 
search of a query having at least one keyword; 

monitoraag feedback from a user as to whether the images in the result set 
are relevant to the^auery; 

in an event tnat the user selects at least one image as being relevant to the 
query, associating the Kl^vword in the query with the selected image to form a first 
keyword-image associationyand assigning a comparatively large weight to the first 
keyword-image association; ah4 

in an event that the user Identifies an example image for refinement of the 
search, associating the keyword iimhe query with the example image to form a 
second keyword-image association ancl assigning a comparatively small weight to 
the second keyword-image association. \ 

21. A method as recited in claim 20, &rther comprising conducting both 
content-based image retrieval and semantic-based rmage retrieval. 

22. A method as recited in claim 20, further^£omprising presenting the 
result set of images in a user interface, the user interfa^ facilitating the user 
feedback by allowing the user to indicate which images ark more relevant and 
which images are less relevant. \ 
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\23. A computer readable medium having computer-executable 
instructions that, when executed on a processor, perform the method as recited in 
claim 20. \ 

24. A method comprising: 

computing, foXeach category, a representative feature vectors of a set of 
existing images within the category; 

determining a set of representative keywords that are associated with the 
existing images in each categiory; 

comparing, for each new^ image, the low-level feature vectors of the new 
image to the representative featurX vectors of the existing images in each category 
to identify a closest matching categoV; and 

labeling the new image with tfte with the set of representative keywords 
associated with the closest matching category. 

25. A method as recited in clain\24, further comprising using use 
feedback to selectively add and/or remove keywOTds from the new image. 

26. A method as recited in claim 24, furthekcomprising: 
placing the labeled new images into a holding caWory; 

evaluating the labeled new images in the holding category to determine if 
any of the keywords associated with the labeled new image match the 
representative keywords from each category; and \ 

assigning the labeled new image to the category that best matches the 
keywords associated with the labeled new image. \ 
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27a An image retrieval system comprising: 
a que^ handler to handle both keyword-based queries having one or more 
search keywords\^nd content-based queries having one or more low-level features 
of an image; and 

a feature and ^mantic matcher to identify at least one of (1) first images 
having keywords that match the search keywords fi*om a kejword-based query, 
and (2) second images having low-level features similar to the low-level features 
of a content-based query. 

28. An image retrieval system as recited in claim 27, wherein the feature 
and semantic matcher ranks the images\ 

29. An image retrieval system as recited in claim 27, wherein the query 
handler comprises a natural language parser. 

30. An image retrieval system as recited in\plaim 27, wherein the query 
handler comprises: 

a parser to parse text-based queries; and 
a concept hierarchy to define various categories of ima^s. 

31. An image retrieval system as recited in claim 27, further comprising 
a user interface to present the images identified by the feature a^d semantic 
matcher. 
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32. An image retrieval system as recited in claim 27, further 
comprisin]g: 

a userViterface to present the images identified by the feature and semantic 
matcher to a uster, the user interface allowing the user to indicate whether the 
images are relevanrto the query; and 

a feedback analyzer to train the image retrieval system based on user 
feedback as to relevancy. 

33. An image retrieval system as recited in claim 27, further 
comprising: 

a user interface to present the iiHfiges identified by the feature and semantic 
matcher to a user, the user interface alP^wing the user to identify an example 
image; and 

the feature and semantic matcher bein^configured to identify additional 
images that contain low-level features similar to tho^se of the example image. 

34. An image retrieval system as recite^ in claim 27, further 
comprising: 

a user interface to present the images identified by theVeature and semantic 
matcher to a user, the user interface allowing the user to identifys^which images are 
relevant to a particular search query; and 

a feedback analyzer to assign a large weight to an associatioi\between the 
search keywords and the images identified as relevant by the user. 
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35. An image retrieval system as recited in claim 34, wherein the 



feedbackv analyzer groups the low-level features of the images identified as 
relevant by\he user. 

36. image retrieval system as recited in claim 27, further 
comprising: 

a user interfa^ to present the images identified by the feature and semantic 
matcher to a user, thk user interface allowing the user to identify an example 
image as being less relevant or irrelevant to the query; and 

a feedback analyzer \p assign a small weight to an association between the 
search ke)words and the exanitole image. 

37. An image retrieval system as recited in claim 36, wherein the feature 
and semantic matcher identifies addit^^onal images with low-level features similar 
to those of the example image. 

38. A database structure stored >^n one or more computer-readable 
media comprising: 

multiple image files; 

multiple keywords; and 

a semantic network to associate the keywords, with the image files, the 
semantic network defining individual keyword-images, links that associate a 
particular keyword with a particular image file, each keywotd-image link having a 
weight indicative of how relevant the particular keyword is to\he particular image 
file. 
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39. computer-readable medium having computer-executable 
instructions that^when executed, direct a computer to: 

find images using both semantic-based image retrieval and low-level 
feature-based image retrieval; 

present the images to\user so that the user can indicate whether the images 
are relevant; and \ 

concurrently conduct semamic-based relevance feedback and low-level 
feature-based relevance feedback. \ 

40. A program as recited in claim jSL further comprising computer- 
executable instructions that, when executed, direct a computer to rank the images. 

41. An information retrieval program, embodied on the computer- 
readable medium, comprising the computer-executable instructions of claim 39. 
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